Electronic blurring compensation device

ABSTRACT

The present invention provides an electronic blurring compensation device applied to a digital camera that includes an angular rate sensor for detecting blurring of an image, an solid-state image pickup element for shooting continuously over time a plurality of images, compensating mutual blurring of the plurality of shot images in accordance with the amount of blurring detected by the angular rate sensor, and generating one image by synthesizing the plurality of compensated images, and a CPU for controlling the number of continuous shootings by the solid-state image pickup element such that the total amount of blurring over the plurality of continuously shot images falls within a previously set predetermined value.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims benefit of Japanese Application No. 2006-047365filed in Japan on Feb. 23, 2006, the contents of which are incorporatedby this reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an electronic blurring compensationdevice for electronically compensating blurring of a shot image.

2. Description of the Related Art

A large number of image pickup devices for picking up still images andmotion images while using a solid-state image pickup element arestructured to be shot by grasping with hand. In such image pickupdevices, for example, when the luminance of a subject is low, theshutter speed becomes slow, and it is known that blurring of images dueto hand movement may easily occur. It is also known that due tovibration during a drive, a camera mounted to a car or the like,blurring of images may similarly occur.

Various technologies for compensating such blurring are proposed up tonow and example of which include an optical blurring compensation forshifting an optical system in accordance with vibration, a sensor shiftblurring compensation for shifting an solid-state image pickup elementin accordance with vibration, and an electronic blurring compensationfor compensating blurring by performing a processing on a picked upimage.

Among these electronic blurring compensation techniques, one correspondsto motion images for keeping the position of a subject in the images bydiffering image cutup positions in accordance to blurring, but thistechnique cannot be applied to still images because the technique is notdesigned for avoiding blurring of an image in one frame.

On the other hand, various electronic blurring compensation techniquesthat can be applied to still images are also proposed.

For example, Japanese Unexamined Patent Application Publication No.2001-45359 describes an image pickup device for subsequently reading outa plurality of images from an image pick up element, storing the imagesin an image memory, and thereafter compensating mutual blurring amongthese plural images to synthesize the images, thereby generating animage in which the blurring has been compensated.

In addition, Japanese Unexamined Patent Application Publication No.2005-198148 describes a solid-state image pickup element including acharge transfer section for horizontally and vertically transferring animage into an image pickup element, in which relative blurring between apicked up first image and a second image that has been transferred tothe charge transfer section is compensated and the first image and thesecond image are synthesized to each other, thereby generating an imagein which the blurring has been compensated.

The technique described in each of the above-mentioned publications isaimed at obtaining one image by performing the synthesis after adjustingthe relative positions of the plurality of images. Incidentally, whenone synthesized image is generated by compensating mutual blurring amongn images (n is a positive integer) which are shot in a time-divisionmanner, an area whose number of images synthesized does not reach n isdeveloped at an end portion of the generated synthesized image. Sucharea whose number of images synthesized does not reach n has graduallysmaller luminance as compared with the area where n images aresynthesized one another, so it is necessary to cut and remove the areaor perform a compensation process on the area. This area whose number ofimages synthesized does not reach n is narrow enough to ignore ascompared with the size of the whole image in a usual case, and thecompensation requires a complicated process. Thus, adoption of the cutand removal method is simple and practical. However, for a device to beused by a large number of general users such as a camera, blurringbecomes extremely large in accordance with a skill of a photographer,and it is also considerable that the effective image area may beunacceptably narrow.

SUMMARY OF THE INVENTION

According to an aspect of the present invention, it is an object toprovide an electronic blurring compensation device with which blurringof an image can be compensated and an effective area of the image largerthan a predetermined size can be secured without depending on atechnique of a photographer.

In brief, according to the present invention, an electronic blurringcompensation device includes: a shooting section for continuouslyshooting a plural images; a detection section for detecting blurring ofthe images; a number-of-shooting control section for controlling thenumber of continuous shootings such that a total amount of blurring overof a predetermined number of continuous images among the plurality ofthe continuously shot images falls within a previously set predeterminedvalue; a blurring compensation section for compensating mutual blurringof the plurality of images shot by the predetermined times on the basisof the control of number-of-shooting control section; and an imagesynthesize section for synthesizing the images compensated by theblurring compensation section.

The above and other objects, features and advantages of the inventionwill become more clearly understood from the following descriptionreferring to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a main electric configuration of a digitalcamera according to a first embodiment of the present invention;

FIG. 2 shows a state where charge accumulated in a photodiode istransferred to a vertical transfer CCD as a first pixel charge accordingto the first embodiment;

FIG. 3 shows a state where charge accumulated in a photo diode afterreading of first pixel charge is transferred to a horizontal transferCCD as second pixel charge and then transferred in a horizontaldirection, and also the first pixel charge is transferred in a verticaldirection according to the first embodiment;

FIG. 4 shows a state in which the first pixel charge is added to thesecond pixel charge according to the first embodiment;

FIG. 5 shows a state where the added charge is saved to a charge holdingsection of the vertical transfer CCD within the same pixel according tothe first embodiment;

FIG. 6 shows a first configuration example of an image pickup elementaccording to the first embodiment;

FIG. 7 shows a second configuration example of the image pickup elementaccording to the first embodiment;

FIG. 8 shows a configuration example of a charge discharge drainaccording to the first embodiment;

FIG. 9 shows another configuration example of a charge discharge drainaccording to the first embodiment;

FIG. 10 is a flowchart showing a process example corresponding to theimage pickup element of the second configuration example shown in FIG. 7when an image is picked up and recorded by a digital camera according tothe first embodiment;

FIG. 11 is a flowchart showing another process example corresponding tothe image pickup element of the second configuration example shown inFIG. 7 when an image is picked up and recorded by a digital cameraaccording to the first embodiment;

FIG. 12 shows an effective area in a superposition relation of threetime-division images according to the first embodiment;

FIG. 13 shows an effective area in an image pickup area of the imagepickup element according to the first embodiment;

FIGS. 14A to 14F are timing charts showing operations of the imagepickup element according to a second embodiment; and

FIG. 15 is a flowchart showing a compensation process for a blurredimage in an information processing section according to the secondembodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Before a detailed description will be given of embodiments, first ofall, a principle for compensating blurring is now described in brief.

It is assumed that an appropriate exposure time (full exposure time)obtained by performing photometry on a subject is 1/15 seconds, forexample. Then, it is also assumed that blur occurs in this exposure timeof 1/15 seconds (shutter speed). On the other hand, when the exposuretime (shutter speed) is 1/125 seconds, no blurring occurs or blurringwhich has by any chance occurred is practically negligible. In such acase, the above-mentioned full exposure time of 1/15 seconds istime-divided into the exposure time of 1/125 seconds to perform theshooting eight times by way of the time-division shooting, and eightimages obtained through this time-division shooting are synthesized(added), thereby obtaining one image in the appropriate exposure time of1/15 seconds. It should be noted that the blurring is not compensated bymerely synthesizing the images obtained through this time-divisionshooting in the above-mentioned 1/125 seconds, so blurring among thetime-division images is mutually compensated before the synthesis. Then,an area where all of the time-division images are overlapped one anotherin this synthesized image functions as an effective area. However, ifthe blurring is too large, this effective area becomes narrow. In viewof the above, in order that the effective area becomes not so narrow,when the amount of blurring is equal to or larger than a predeterminedacceptable amount, an image exceeding this acceptable level is excludedfrom the synthesis target. Then, a signal level of the synthesized imageobtained by even excluding one or more time-division images does notreach a predetermined level as it is and therefore in accordance withthe number of time-division images excluded, that is, in accordance withthe number of synthesized time-division images, the image is amplifiedto obtain an appropriate signal level.

According to a first embodiment to be described below, theabove-mentioned blurring compensation and synthesis of time-divisionimages are performed in the image pickup element at a high speed. Then,according to a second embodiment to be described below, theabove-mentioned blurring compensation and synthesis of the time-divisionimages are performed after the time-division images are read out fromthe image pickup element to be converted into digital signals.

Hereinafter, the embodiments of the present invention will be describedwith reference to the drawings.

First Embodiment

FIGS. 1 to 13 show a first embodiment of the present invention and FIG.1 is a block diagram of a main electric configuration of a digitalcamera. According to this embodiment, an electronic blurringcompensation device is applied to a digital camera.

This digital camera includes a solid-state image pickup element(hereinafter abbreviated as image pickup element when appropriate) 1, acorrelated double sampling (CDS) circuit 2, a gain controller amplifier(AMP) 3, an A/D converter 4, a timing generator (TG) 5, a signalgenerator (SG) 6, a CPU 7, an information processing section 8, a DRAM9, a compress/expand section 10, a recording medium 11, a liquid crystaldisplay section 12, an interface section 13, a lens driver system 14, ashooting lens 15, an aperture driver system 16, an aperture 17, a firstrelease switch 1 8a and a second release switch 1 8b, an angular ratesensor 19 and an angular rate sensor 20, an A/D converter 21 and an A/Dconverter 22, a distance detection section 23, an EEPROM 24 built in theCPU 7, a shooting mode setting section 25, and a shooting conditionsetting section 26.

The shooting lens 15 is an image pickup optical system for forming asubject image on an image pickup surface of the image pickup element 1and structures a shooting section.

The aperture 17 is an optical aperture for performing light amountadjustment by regulating a passing range of image formation luminousflux from the shooting lens 15. The aperture 17 is a part of the imagepickup optical system and structures the shooting section.

The image pickup element 1 is adapted to photoelectrically convert thesubject image formed via the aperture 17 by the shooting lens 15 to beoutputted as electric signals. The image pickup element 1 constructs theshooting section. Herein, FIG. 6 shows a first configuration example ofthe image pickup element 1 and FIG. 7 shows a second configurationexample of the image pickup element 1. The image pickup elements 1respectively shown in FIGS. 6 and 7 both have a larger number of photodiodes arranged in matrix, a horizontal transfer CCD or a verticaltransfer CCD functioning as a first register for holding a first imagegenerated by the photo diodes through exposure control at such a highspeed that blurring can be tolerated, and a horizontal transfer CCD or avertical transfer CCD functioning as a second register that is differentfrom the first register for holding a second image shot after the firstimage. In each of the image pickup elements, in order that mutualblurring between the two images held in the first register and thesecond register is cancelled, such an operation will be repeatedlyperformed that after these two images are shifted in the first registerand the second register, the two images are synthesized to each other,and this synthesized image is held at the first register or the secondregister, thereby generating the blurring compensated image in the imagepickup element 1. The detailed configuration and action of the imagepickup element 1 will be described later.

The TG 5 is adapted to supply a transfer pulse for driving the imagepickup element 1 and constitutes a shooting section and a signalprocessing section.

The CDS 2 is driven in accordance with a sample hold pulse that issupplied from the TG 5 and performs correlated double sampling or thelike on an image signal outputted from the image pickup element 1 toremove a reset noise therefrom. The CDS 2 constitutes the shootingsection and the signal processing section.

The SG 6 is adapted to generate a synchronous signal on the basis of thecontrol of the CPU 7 to be outputted to the TG 5. Also the SG 6constitutes the shooting section and the signal processing section.

The gain controller amplifier (AMP) 3 is an amplification section foramplifying an analog signal outputted from the CDS 2. The gaincontroller amplifier also constitutes the shooting section and thesignal processing section. An amplification gain of the gain controlleramplifier (AMP) 3 is set to an amplification gain in accordance with anISO (International Organization for Standardization) sensitivity Sv, inother words, the gain controller amplifier (AMP) 3 functions as an ISOsensitivity changing section. In addition, the amplification gain of thegain controller amplifier (AMP) 3 is used also in the case where thesynthesized image is amplified for offsetting a shortage when the numberof images obtained through this time-division shooting does not reach aregulated number.

The A/D converter 4 is an analog/digital converting section forconverting an analog signal outputted from the gain controller amplifier(AMP) 3 in accordance with a signal supplied from the TG 5. The A/Dconverter 4 constitutes the shooting section and the signal processingsection.

The information processing section 8 is adapted to generate image databy performing a process on a pixel signal outputted from the A/Dconverter 4. The information processing section 8 constitutes theshooting section, a blurring compensation section, an image synthesizingsection, the signal processing section. The information processingsection 8 includes an effective area extraction section having afunction of extracting an image data in which blurring is appropriatelycompensated from image data outputted from the image pickup element 1.Furthermore, the information processing section 8 is configured toinclude a buffer memory 8 a functioning as a memory section fortemporarily storing the image data.

The DRAM 9 is adapted to temporarily store the image data outputted fromthe information processing section 8 and also temporarily store imagedata obtained by expanding compressed image data read out from therecording medium 11 by the compress/expand section 10. It should benoted that such a configuration may be adopted that the buffer memory 8a doubles as the function of the DRAM 9.

The compress/expand section 10 is adapted to compress the image datastored in the DRAM 9 and expand the compressed image data read out fromthe recording medium 11.

The recording medium 11 is a recording section for recording the imagedata compressed by the compress/expand section 10. For example, therecording medium 11 is made of a non-volatile recording medium.

The liquid crystal display section 12 is adapted to display the imagedata outputted from the information processing section 8 or the expandedimage data outputted from the DRAM 9. The liquid crystal display section12 doubles as a display section for displaying various warning messagesand the like.

The interface section 13 is an interface including a terminal fortransmitting and receiving data with an external device such as amonitor or a personal computer. Via the interface section 13, the imagedata or the like supplied from the information processing section 8 orthe DRAM 9 can be outputted to the external device. In some cases, imagedata or the like can be taken in from the external device into thedigital camera.

The lens driver system 14 is adapted to drive the shooting lens 15 to afocal position by receiving the instruction from the CPU 7 on the basisof an object distance detected by the distance detection section 23. Theabove-mentioned process is known as so-called auto-focus control. Itshould be noted herein that the auto-focus control is conducted on thebasis of the output from the distance detection section 23, but such astructure may be adopted. The CPU 7 extracts high frequency componentswith use of a bypass filter from the luminance component of the imagedata for one frame (one screen) stored in the DRAM 9, an accumulatedsynthesized value of the extracted high frequency components iscalculated or the like, an AF evaluation value corresponding to thecontour component on the high frequency side or the like is calculated,and focal point detection is performed on the basis of the AF evaluationvalue.

The aperture driver system 16 is an aperture control section for drivingthe aperture 17 to change the opening diameter while the CPU 7functioning as a photometry section performs exposure calculation on thebasis of the image data stored in the DRAM 9 and receives an instructionbased on the result from the CPU 7. Such a process is known as so-calledAE (automatic exposure) control.

The angular rate sensor 19 is a blurring detection section for detectingan angular rate when the digital camera is rotated with an X axisdirection as a rotation center while the right hand side in thehorizontal directions is set as the X axis direction as the digitalcamera is viewed from the object side.

On the other hand, the angular rate sensor 20 is a blurring detectionsection for detecting an angular rate when the digital camera is rotatedwith a Y axis direction as a rotation center while the upper side in thevertical directions is set as the Y axis direction.

The A/D converter 21 is adapted to convert an analog signal indicatingthe angular rate detected by the angular rate sensor 19 into a digitalsignal at a predetermined time interval (sampling interval). The A/Dconverter 21 is a part of the blurring detection section.

In a similar manner, the A/D converter 22 is adapted to convert ananalog signal indicating the angular rate detected by the angular ratesensor 20 into a digital signal at the predetermined time interval(sampling interval). The A/D converter 21 is also a part of the blurringdetection section.

The CPU 7 performs time integration on the digital signal converted bythe A/D converter 21. This digital signal subjected to the timeintegration corresponds to the amount of rotation with the X axis of thecamera main body as the rotation center. Then, whether the rotationdirection about the X axis is clockwise or counter-clockwise isdetermined depending on a positive analog output signal or a negativeanalog output signal of the angular rate sensor 19.

In a similar manner, the CPU 7 performs time integration on the digitalsignal converted by the A/D converter 22. This digital signal subjectedto the time integration corresponds to the amount of rotation with the Yaxis of the camera main body as the rotation center. Then, whether therotation direction about the Y axis is clockwise or counter-clockwise isdetermined depending on a positive analog output signal or a negativeanalog output signal of the angular rate sensor of the angular ratesensor 20.

The first release switch 18 a is a first stage of a release switch madeof automatic returning type two stages for inputting an instruction ofthe image pickup operation. When the release switch is pressed down andthe first release switch 18 a is turned on, a distance measurement andthe photometry operation are performed.

The second release switch 18 b is a second stage of a release switchmade of automatic returning type two stages for inputting an instructionof the image pickup operation. When the release switch is pressed downfurther and the second release switch 18 b is turned on, the imagepickup element 1 performs the image pickup operation, thereby generatingthe image data in the above-mentioned manner. After being compressed,the image data is recorded in the recording medium 11.

The distance detection section 23 is adapted to detect a distance to thesubject and can appropriately adopt a known structure.

The shooting mode setting section 25 is adapted to select one of ashutter priority shooting mode, an aperture priority shooting mode, anda program shooting mode.

The shooting condition setting section 26 is adapted to change variousshooting conditions such as a shutter speed (exposure time), an aperturevalue, and an ISO sensitivity.

The CPU 7 incorporates the EEPROM 24 as a non-volatile memory forstoring a relation among an exposure value Ev, Tv for performing theexposure control in an optimal way (apex value of the exposure time),and Av (apex value of the aperture value) as a program chart. The EEPROM24 can appropriately store other information necessary to the digitalcamera.

A signal from the first release switch 18 a, a signal from the secondrelease switch 18 b, a signal from the angular rate sensor 19 via theA/D converter 21, a signal from the angular rate sensor 20 via the A/Dconverter 22, a signal from the shooting mode setting section 25, and asignal from the shooting condition setting section 26 are inputted tothe CPU 7. Then the CPU 7 is adapted to output instructions to the TG 5and the SG 6.

Furthermore, the CPU 7 is connected to the information processingsection 8, the DRAM 9, the lens driver system 14, the aperture driversystem 16, and the distance detection section 23 in a bidirectional wayand functions as a control section for controlling the entire digitalcamera including those parts. Also the CPU 7 doubles as the shootingsection, the blurring detection section, a shooting number controlsection, a blurring compensation section, and the image synthesizingsection.

To be specific, the CPU 7 is adapted to perform the above-mentionedauto-focus control as well as the AE control and switching of the drivemode of the image pickup element 1 based on signals for instructingtaking in of still images from the first release switch 18 a and thesecond release switch 18 b. Furthermore, the CPU 7 is also adapted toperform a control for changing the opening of the aperture 17, anexposure time control on the image pickup element 1, and the like. Then,the CPU 7 sets a shooting mode of the digital camera on the basis of aninput from the shooting mode setting section 25 and sets a shootingcondition of the digital camera on the basis of an input from theshooting condition setting section 26. In addition, the CPU 7 is alsoadapted to perform a calculation for the amount of blurring and the likeon the basis of outputs from the angular rate sensors 19 and 20.

Next, with reference to FIGS. 2 to 5, an operation principle of theimage pickup element 1 will be described. FIG. 2 shows a state wherecharge accumulated in a photodiode is transferred to a vertical transferCCD as a first pixel charge, FIG. 3 shows a state where chargeaccumulated in a photo diode after reading of first pixel charge istransferred to a horizontal transfer CCD as second pixel charge and thentransferred in a horizontal direction, and also the first pixel chargeis transferred in a vertical direction, FIG. 4 shows a state in whichthe first pixel charge is added to the second pixel charge, and FIG. 5shows a state where the added charge is saved to a charge holdingsection of the vertical transfer CCD within the same pixel.

In the image pickup element 1, a plurality of photo diodes 27 forgenerating charge by receiving light beam from the subject are arrangedin matrix. The photo diodes 27 arranged in matrix constitute aphotoelectric conversion section.

Moreover, the image pickup element 1 includes horizontal transfers CCD28 that are arranged adjacent in rows of the photo diodes 27 in thephotoelectric conversion section and function as the horizontal transferregister, the blurring compensation section, and the synthesizingsection. The horizontal transfer CCD 28 is adopted to store the firstimage obtained by reading the charge generated in the photo diodes 27and transfers the first image in the horizontal direction.

In addition, the image pickup element 1 includes vertical transfers CCD29 that are arranged adjacent in columns of the photo diodes 27 in thephotoelectric conversion section and function as the vertical transferregister, the blurring compensation section, and the synthesizingsection. The vertical transfer CCD 29 is adapted to store the secondimage according to the synthesized charge generated by synthesizing thecharges obtained before the first image is obtained and transfers thesecond image in the vertical direction.

Then, an electrode part arranged at an intersection position of thehorizontal transfer CCD 28 and the vertical transfer CCD 29, functionsas the synthesizing section for synthesizing the first image and thesecond image in an analog manner.

It should be noted herein that the example is shown in which theplurality of photo diodes 27 are arranged in the lengthwise directionand in the crosswise direction perpendicular to the lengthwise directionto form a matrix, but the configuration is not limited to the above aslong as an arrangement forms a substantial matrix. For example, theplurality of photo diodes 27 may be arranged in one direction and theother direction oblique to the one direction to form a matrix. At thistime, the horizontal transfer CCD 28 and the vertical transfer CCD 29may be arranged in directions to be obliquely intersected. Furthermore,the shape of the nhoto diode 27 is not limited to square or rectangle,and may have other various shapes of parallelogram, triangle, andhexagon.

One pixel 38 in the image pickup element 1 includes one of the photodiodes 27, and a part of the horizontal transfer CCD 28 and a part ofthe vertical transfer CCD 29 which are adjacent to the photo diode 27.Then, the size of the one pixel 38 is as follows. The length of thehorizontal direction (crosswise direction) is set as Lx, and the lengthof the vertical direction (lengthwise direction) is set as Ly.

Operation of the image pickup element 1 with such a structure will bedescribed.

Hereinafter, the photo diode arranged at the left top corner of theimage pickup element 1 is denoted by “P1, 1” and the photo diodearranged at an i-th on the right hand side in the horizontal direction(i is an integer number equal to or larger than 1) and at an j-th on thelower side in the vertical direction (j is an integer number equal to orlarger than 1) is denoted by “Pi, j”.

FIG. 2 shows a state where the charge of the photo diode Pi, j which issubjected to the first photoelectric conversion (first pixel charge)(which is represented by a circle mark in the drawing) is shifted (readout) to the vertical transfer CCD 29 adjacent to the photo diode Pi, j.It should be noted that in FIG. 2, only the first pixel charge relatedto the photo diode Pi, j is shown but other charges accumulated throughthe photoelectric conversion in all the other photo diodes for the sameperiod of time are also shifted in a similar manner to the verticaltransfers CCD 29 all at once.

FIG. 3 shows a state where the charge of the photo diode Pi-1, j-1accumulated through the photoelectric conversion immediately after thefirst pixel charge is shifted (second pixel charge) is firstly shifted(read out) to the horizontal transfer CCD 28 adjacent to the photo diodePi-1, j-1. It should be noted that in FIG. 3 as well, only the secondpixel charge related to the photo diode Pi-1, j-1 is shown but othercharges accumulated through the photoelectric conversion in all theother photo diodes for the same period of time are also shifted in asimilar manner to the horizontal transfer CCD 28 all at once. Herein,the light from the subject reaching the nhoto diode Pi.j at the time ofthe first pixel charge accumulation is shifted to a position reachingthe photo diode Pi-1,j-1 at the time the second pixel chargeaccumulation because of blurring of images due to hand movement or thelike. The change in the reaching position of this light can be found outonly when the accumulation of the second pixel charge has beencompleted. Thus, in the state shown in FIG. 2, the first pixel charge isnot yet transferred and is only held (stored) in the vertical transferCCD 29. However, after reading out the second pixel charge, the changein the reaching position of the light is found out on the basis of theoutputs from the angular rate sensors 19 and 20, the positional relationbetween the first pixel charge and the second pixel charge related tothe same subject light is obtained and in order to conduct a synthesisto be described later, the charges are transferred to the positionsadjacent to each other (position in the same one pixel 38). That is,FIG. 3 shows an example in which the second pixel charge is transferredby one pixel on the horizontal transfer CCD 28 in the right hand side onthe paper surface, and also the first pixel charge is transferred by onepixel on the vertical transfer CCD 29 in the upper side on the papersurface. It should be noted that the transfer of the first pixel chargeand the transfer of the second pixel charge are performed for the pixelcharges of all the photo diodes as described in the above. In order toperform the above-mentioned transfer of the horizontal transfer CCD 28and transfer of the vertical transfer CCD 29, it is necessary toappropriately arrange transfer electrodes to avoid interference betweenthe charge in the horizontal transfer CCD 28 and the charge in thevertical transfer CCD 29 at the positions intersected by the respectivetransfer CCDs.

FIG. 4 shows a state where the first pixel charge and the second pixelcharge are shifted to the intersection position between the horizontaltransfer CCD 28 and the vertical transfer CCD 29, and the synthesis isconducted at the intersection position (the synthesis is represented by“+” in FIG. 4). It is needless to mention that this synthesis isperformed for the first pixel charges to all the photo diodes and thesecond synthesized charges related to all the photo diodes.

As a result, the first image (image composed of all the first pixelcharges) and the second image (image composed of all the second pixelcharges) continuously shot immediately after the first image is shot areshifted by the amount of blurring, in other words, after the blurring iscompensated, the synthesis is conducted.

It should be noted that when the third or after time-division image isnewly read out, the first image corresponds to a synthesized imageobtained by synthesizing the previous time-division images (which isobtained by subsequently compensating the blurring and synthesizing thetime-division images from the first time-division image to thetime-division image one before the latest image). Thus, by performingthe operations shown in FIG. 4, the synthesis is conducted after therelative amount of blurring between the new time-division image (imagecomposed of all the new pixel charges) and the synthesized image iscompensated.

FIG. 5 shows a state where for example, the pixel charge synthesized atthe intersection position between the horizontal transfer CCD 28 and thevertical transfer CCD 29 is transferred (saved) to the charge holdingsection of the vertical transfer CCD 29 in the same pixel. Theintersection position is used for both the horizontal transfer and thevertical transfer, and therefore if the synthesized charge is held atthe intersection position, the synthesis with the pixel charge read outnext cannot be conducted.

In view of the above, herein, the charge after the synthesis is oncesaved to the charge holding section of the vertical transfer CCD 29. Asa result, when the next pixel charge is read out to the horizontaltransfer CCD 28, as in the same manner described above, the pixelsynthesis can be performed.

It should be noted herein that, the charge after the synthesis istransferred (saved) to the charge holding section of the verticaltransfer CCD 29 in the same pixel, but instead of this, the charge afterthe synthesis may be transferred (saved) to the charge holding sectionof the horizontal transfer CCD 28 in the same pixel. At this time, thenext pixel charge is read out to the vertical transfer CCD 29. Also, thepixel charge after the synthesis is not necessarily saved within thesame pixel.

Therefore, other than the examples shown in FIGS. 2 to 5, thesynthesized charge may be stored in one of the memory sections of thehorizontal transfer CCD 28 and the vertical transfer CCD 29, and the newpixel charge may be read out and stored in the other memory section ofthe horizontal transfer CCD 28 or the vertical transfer CCD 29.

As described in the above, the following sequence is repeatedlyperformed.

Image shift to the horizontal transfer CCD

-   -   Charge transfer for relative blurring compensation    -   Synthesis of the charges    -   Saving of the synthesized charge from the intersection position        between the horizontal transfer CCD and the vertical transfer        CCD. It should be noted that the time-division image of the        first time-division shooting among the plurality of        time-division shootings has the amount of blurring of 0 and the        synthesized value of the vertical transfer CCD of 0, so it        suffices that the similar sequence is executed.

It should be noted that FIGS. 2 to 5 show the example in which the newtime-division image is shifted by one pixel in the left-hand side andone pixel in the upper direction with respect to the synthesized image,but in general the new time-division image is shifted in the horizontaldirection and the vertical direction by the appropriate number of pixelsaccording to the about of blurring.

The operation principle of the blurring compensation as described in theabove is for compensating the relative blurring in the X and Ydirections between the two image accumulated in the horizontal transferCCD and the vertical transfer CCD which are arranged while surroundingthe respective photo diodes of the image pick up section (refer to FIG.6). On the other hand, as will be described with reference to FIG. 7, anaccumulation section for accumulating an image is provided separatelyfrom the image pickup section in the image pickup element 1. Then, thevertical transfer CCD for holding the image obtained through thistime-division shooting and the horizontal transfer CCD for holding thesynthesized image obtained through the synthesis after the blurringcompensation between this time-division image and the previoustime-division are provided in the accumulation section. In this case aswell, in a similar manner, the image in which blurring is compensated inthe image pickup element 1 can be generated.

Next. FIG. 6 shows a first configuration example of the image pickupelement 1.

The image pickup element 1 according to the first configuration exampleis structured to include, as shown in FIG. 6, the photo diodes 27 forsubjecting the image to the photoelectric conversion for accumulation,the horizontal transfer CCD 28 for transferring the charges read outfrom the photo diodes 27 in the horizontal direction, the verticaltransfer CCD 29 for transferring the charges read out from the photodiodes 27 in the vertical direction, a charge release drain 30 forreleasing the charge transferred to an end part of the horizontaltransfer CCD 28 or the vertical transfer CCD 29 to the outside of atransfer path, and a reading horizontal transfer CCD 31 for reading outthe charge transferred from the vertical transfer CCD 29 from the imagepickup element 1 to the outside.

Herein, the charge release drains 30 are arranged to form a U shapealong the periphery of the image pickup section except the readinghorizontal transfer CCD 31 side so that the charges transferred to bothend parts of all the horizontal transfers CCD 28 and an end parts on theupper side of the paper surface in all the vertical transfers CCD 29 ofFIG. 6 can be released to the outside of the transfer path.

In such a configuration, the pixel charge accumulated through thephotoelectric conversion by the photo diodes 27 is read out to thehorizontal transfer CCD 28 and then accumulated therein. The chargeaccumulated in the horizontal transfer CCD 28 forms the first image as awhole. Then, the vertical transfer CCD 29 stores the previoussynthesized charge. The synthesized charge forms the second image as awhole. The newly read pixel charge is transferred by the shift amountcalculated on the basis of the output of the angular rate sensor 20 onthe basis of the blurring about the Y axis for compensating the relativeamount of blurring in the horizontal direction with respect to thesecond image stored in the vertical transfer CCD 29 in the readhorizontal transfer CCD 28. On the other hand, the second image that isthe synthesized image stored in the vertical transfer CCD 29 istransferred by the shift amount calculated on the basis of the blurringabout the X axis calculated on the basis of the output of the angularrate sensor 19 for compensating the relative amount of blurring in thevertical direction with respect to the first image stored in thehorizontal transfer CCD 28 in the vertical transfer CCD 29.

In this way the shift by the amount for compensating the blurring isperformed, and thereafter, at the intersection position between thehorizontal transfer CCD 28 and the vertical transfer CCD 29, the newlyread out pixel charge and the previous synthesized charge aresynthesized to be stored in the vertical transfer CCD 29, therebygenerating a new synthesized charge in which the relative blurring iscompensated. The operations described above are performed by thepredetermined number of times set as the number of times for performingthe time-division shooting, for example, 10 times. Herein, the chargesaccumulated in the vertical transfer CCD 29 and the horizontal transferCCD 28 are shifted in the upper, lower, left, and right according to theblurring and therefore the charge reaching the end part of therespective transfer CCDs 28 and 29 is released via the charge releasedrain 30 to the outside of the transfer path.

In the first configuration example shown in FIG. 6, the charge releasedrain 30 is configured as shown in FIG. 8. Herein, FIG. 8 shows aconfiguration example of the charge release drain 30. In theconfiguration example shown in FIG. 8, on a surface of an n-typesubstrate (for example, n-type silicon substrate) 42, first of all, ap-type diffused area (p-well)43 and an n-type diffused area 44 areformed in the stated order toward the front surface side. Furthermore,an n+ diffusion area drain 30 is formed so as to be adjacent to apotential well (n− diffusion area 44) located under a transfer electrode39 that is the end part of the horizontal transfer CCD 28 or thevertical transfer CCD 29. Then, the charge reaching the transferelectrode 39 that is the end part is released via the n+ diffusion areadrain 30. On the other hand, the charge reaching the end part of thereading horizontal transfer CCD 31 side of the vertical transfer CCD 29is released via the reading horizontal transfer CCD 31. With theprovision of such a charge release mechanism, it is possible to avoidthe overflow of the charge at the horizontal transfer CCD 28 and thevertical transfer CCD 29.

The thus finally obtained synthesized charge functions as a chargeconstitute an image in which blurring is compensated.

It should be noted that in the first configuration shown in FIG. 6, thetime-division images are read out to the horizontal transfer CCD 28 andthen the synthesized image is held at the vertical transfer CCD 29, butinstead of this construction, it is also possible that the time-divisionimages are read out to the vertical transfer CCD 29 and the synthesizedimage is held at the horizontal transfer CCD 28.

Next, FIG. 7 shows the second configuration example of the image pickupelement 1.

The image pickup element shown in FIG. 7 is a so-called FIT type CCD(frame interline transfer CCD) in which the image pickup section 1 a andan accumulation section 1 b are separated from each other.

The image pickup section 1 a is structured to include a photo diode 32for accumulating an image through photoelectric conversion and avertical transfer CCD 33 for transfer a charge read out from the photodiode 32 in the vertical direction.

The accumulation section 1 b is structured to include a verticaltransfer CCD 34 for transferring the charge transferred from thevertical transfer CCD 33 of the image pickup section 1 a to apredetermined position in the accumulation section 1 b, a horizontaltransfer CCD 35 arranged so as to intersect the vertical transfer CCD34, a charge release drain 36 for releasing the charge transferred toboth ends of the horizontal transfer CCD 35 to the outside of thetransfer path, a charge release drain 41 for releasing the chargetransferred to the end part of the image pickup section la side of thevertical transfer CCD 34 to outside of the transfer path, and a readinghorizontal transfer CCD 37 for reading out the charge transferred fromthe vertical transfer CCD 34, to the outside from the image pickupelement 1.

In the thus configured image pickup element 1, the pixel chargeaccumulated through the photoelectric conversion by the photo diode 32is transferred via the vertical transfer CCD 33 to the vertical transferCCD 34 of the accumulation section 1 b. The charge transferred to thevertical transfer CCD 34 forms the first image as a whole

First of all, the first image transferred to the vertical transfer CCD34 through the first time-division shooting is then transferred to thehorizontal transfer CCD 35. This image accumulated at the horizontaltransfer CCD 35 is set as a second image.

The image obtained through the next time-division shooting is againtransferred to the vertical transfer CCD 34. Then the first image isshifted by a predetermined amount in a direction for cancelling therelative deviation in the vertical direction with respect to the secondimage accumulated at the horizontal transfer CCD 35 on the basis of theblurring about the X axis that is calculated on the basis of the outputfrom the angular rate sensor 19. Then, the second image is shifted by apredetermined amount in a direction for cancelling the relativedeviation in the horizontal direction with respect to the first imageaccumulated at the vertical transfer CCD 34 on the basis of the blurringabout the Y axis that is calculated on the basis of the output from theangular rate sensor 20. Then, the first image and the second image thatare shifted so as to cancel the relative blurring in the verticaldirection and the horizontal direction are synthesized at theintersection position between the vertical transfer CCD 34 and thehorizontal transfer CCD 35 or in the vicinity thereof before beingaccumulated at the horizontal transfer CCD 35.

After that, the time-division shooting is conducted again, an imageobtained by this shooting is held at the vertical transfer CCD 34 of theaccumulation section 1 b as the first image. Then, the synthesized imagein which the blurring is compensated in a similar manner is generated tobe held at the horizontal transfer CCD 35. Such operations are performedby a predetermined number of times set as the number of times forperforming the time-division shooting, for example, 10 times. Herein,the charges accumulated at the vertical transfer CCD 34 and thehorizontal transfer CCD 35 are shifted according the blurring in theupper, lower, left, and right directions, and therefore the chargereaching the end part of the horizontal transfer CCD 35 is released viathe charge release drain 36, the charge reaching the end part of theimage pickup section 1 a side of the vertical transfer CCD 34 isreleased via the charge release drain 41, and the charge reaching theend part of the reading horizontal transfer CCD 37 side of the verticaltransfer CCD 34 is released via the reading horizontal transfer CCD 37,to the outside of the transfer path.

The charge release drain 36 in the image pickup element 1 according tothe second configuration example shown in FIG. 7 functions as ann+diffusion area drain provided so as to be adjacent to the potentialwell (n− diffusion area) located under the transfer electrode that isthe end part of the horizontal transfer CCD 35, similarly to the chargerelease drain 30 (refer to FIG. 8) in the image pickup element 1according to the first configuration example shown in FIG. 6.

On the other hand, if the charge release drain 41 in the image pickupelement 1 according to the second configuration example is configuredsimilarly to that of FIG. 8, the charge from the image pickup section 1a is not transferred to the accumulation section 1 b. In view of theabove, the charge release drain 41 is configured as shown in FIG. 9.Herein, FIG. 9 shows a configuration example of the charge release drain41. That is, a gate electrode 40 is provided so as to be adjacent to thetransfer electrode 39 that is the end part of the image pickup section 1a side of the vertical transfer CCD 34 and the charge release drain 41that is made of the n+ diffusion area is arranged at the positionopposed to the transfer electrode 39 with the gate electrode 40interposed therebetween. Then, the charge transferred to the end part ofthe image pickup section 1 a side of the vertical transfer CCD 34 isselectively released via the gate electrode 40 to the charge releasedrain 41.

That is, when the charge from the image pickup section 1 a istransferred to the accumulation section 1 b, the gate electrode 40 isapplied with a low voltage. As a result, the charge transferred throughthe vertical transfer CCD 34 is prevented from being released to thecharge release drain 41. On the other hand, when the charge accumulatedat the accumulation section 1 b is transferred in the vertical transferCCD 34 for compensating the blurring, the gate electrode 40 is appliedwith a high voltage, whereby the charge transferred to the end part ofthe image pickup section la side of the vertical transfer CCD 34 isreleased to the charge release drain 41. Then, the charge reaching theend part on the reading horizontal transfer CCD 37 side of the verticaltransfer CCD 34 is released via the reading horizontal transfer CCD 37.

With such a configuration and actions, the charge can be prevented fromoverflowing at the vertical transfer CCD 34 and the horizontal transferCCD 35.

It should be noted that in the above-mentioned configuration example, atthe time of charge transfer for the blurring compensation, the charge ofthe end part of the image pickup section 1 a side of the verticaltransfer CCD 34 is released to the charge release drain 41. However,instead of this configuration, at the time of charge transfer for theblurring compensation, the vertical transfer CCD 33 may be driven at thesame time to transfer (release) the charge reaching an upper end part ofthe vertical transfer CCD 34 to the vertical transfer CCD 33, therebypreventing the charge overflow. Then, in this case, the charge releasedto the vertical transfer CCD 33 is released via the vertical transferCCD 34 (the vertical transfer CCD 34 without holding the charge in astate where the synthesis of the first image and the second image isperformed to be accumulated at the horizontal transfer CCD 35) and thereading horizontal transfer CCD 37, and thereafter the nexttime-division shooting may be preformed.

When the blurring compensation and synthesis operations are completed bythe predetermined of times as described above, the image held at thehorizontal transfer CCD 35 is transferred to the vertical transfer CCD34 and then read out to the outside via the vertical transfer CCD 34 andthe reading horizontal transfer CCD 37 from the image pickup element 1.

Next, FIG. 10 is a flowchart showing a process example corresponding tothe image pickup element 1 of the second configuration example shown inFIG. 7 when an image is picked up by a digital camera.

When a power source of the digital camera is turned on (for example, abattery is exchanged) or an operation start switch not shown in thedrawing (for example, a power source switch) is operated, an operationof this digital camera is started.

Once a process is started, after a predetermined initiative valuesetting or the like is conducted, first of all, it is determined whetheror not the first release switch 18 a is in a closed state in response toa release operation of a photographer (Step S101).

Herein, in the case where the first release switch 18 a is not in theclosed state, the process is branched to J101, the detection of thefirst release switch 18 a is similarly repeatedly performed. It shouldbe noted that in actuality, a display is conducted or such an operationfor detection of an input of a key not shown is performed between J101and Step S101, but a description of the above-mentioned generaloperation will be appropriately omitted below as well.

In Step S101, when it is detected that the first release switch 18 a isin the closed state, next, the blurring limit exposure time T_(Limit) iscalculated (Step S102). The blurring limit exposure time T_(Limit) is anassumed period of time during which the amount of blurring from thestart of exposure reaches an allowable limit amount of blurring.

Now the blurring limit exposure time T_(Limit) will be described. As alongtime empirical rule related to so-called Leica frame (also known as:double frame) camera of 24 mm height×36 mm width (43.28 mm diagonal) ina 35 millimeter film camera, it is known that when a focal length of ashooting lens in unit of the millimeter is set as f, the blurring limitexposure time T_(Limit) becomes T_(Limit)≈1/f (second). In thisembodiment, this empirical rule is applied in consideration with a sizeof a shooting image frame set in an effective image pickup area of theimage pickup element 1 of the digital camera. It should be noted thatthe blurring limit exposure time T_(Limit) does not necessarily use avalue given on the basis of 1/f, and to be brief, such an exposure timemay be used with which the blurring is not substantially generated.Therefore, the blurring limit exposure time T_(Limit) may be set as atime shorter than the exposure time given on the basis of 1/f ingeneral.

Next, photometry is performed on the brilliance of the subject (Step S103). This photometry is adapted to monitor the levels of image signalsrepeatedly outputted from the image pickup element 1 so that thebrilliance of the subject is calculated. That is, the image signals readfrom the image pickup element 1 is processed by the CDS 2 and amplifiedby the gain controller amplifier 3. Then the signals are converted todigital values by the A/D converter 4, passed through the informationprocessing section 8, and temporarily stored in the DRAM 9. Among theimage signals stored in the DRAM 9, the image signal in a predeterminedarea in the vicinity of the center section in the entire image, forexample, is read out by the CPU 7 to find out addition average value atthe level. Then, the CPU 7 calculates a brilliance of the subject (Bv)on the basis of the thus found out addition average value.

Subsequently, the CPU 7 calculates a shutter speed value (exposure time)T_(Exp) and the aperture value of the aperture 17 which are necessary toobtain the appropriate exposure and also performs the aperture settingof the aperture 17 on the basis of the calculation results via theaperture driver system 16 (Step S104). Herein, the exposure time T_(Exp)has a relation of T_(Exp)=m×ΔT_(Exp) when the exposure time of thetime-division shooting is set as ΔT_(Exp) and the number of times forperforming the time-division shooting is set as m.

Next, it is determined whether or not the second release switch 18 b isin the closed state (Step S105). Herein, when the second release switch18 b is not in the closed state, as long as the first release switch 18a is in the closed state, the process is branched to J102, and while theabove-mentioned processes in Steps S102 to S105 are repeatedlyperformed, a moment is waited for when the second release switch 18 b isput into the closed state.

In this way, in Step S105, when it is determined that the second releaseswitch 18 b is in the closed state, it is determined whether or not theexposure time T_(Exp) is shorter than the blurring limit exposure timeT_(Limit) (Step S106).

In this Step S106, when it is not determined that T_(Exp)<T_(Limit),next, a value obtained by dividing the exposure time T_(Exp) by thenumber of times m for performing the time-division shooting is stored ina memory for storing the exposure time ΔT_(Exp) of the time-divisionshooting (Step S107). It should be noted that [] means a memory thatstores data in brackets. Therefore, [ΔT_(Exp)] means a memory thatstores a variable ΔT_(Exp) in brackets.

Next, an initial value “0” is stored in a memory [n] that stores thenumber of times n for actually performing the time-division shooting(Step S108). Herein, as the exposure time of the time-division shootingΔT_(Exp), there are a method of using the blurring limit exposure timeT_(Limit) and a method of using the exposure time obtained by dividingthe exposure time T_(Exp) calculated in Step S103 by the previously setnumber of times m for performing the time-division shooting. The methodof using the blurring limit exposure time T_(Limit) as the exposure timeof the time-division shooting ΔT_(Exp) is superior in compensating theblurring with certainty but if the exposure time T_(Exp) becomes long,the number of times for performing the time-division shooting isincreased. If the number of times for performing the time-divisionshooting is increased, the signal amount obtained through one time ofthe time-division shooting becomes small, and therefore S/N (signal tonoise ratio) may become low. Then, the saturation signal amount of thephoto diode needs to be adjusted in accordance with the number of timesfor performing the time-division shooting, and thus the configuration ofthe image pickup element becomes complicated. Therefore, it is betterwhen the type of the number of times m for performing the time-divisionshooting is as few as possible. In view of the above, in thisembodiment, the number of times m for performing the time-divisionshooting has only one type. Then, the exposure time of the time-divisionshooting uses a value obtained by dividing the exposure time T_(Exp) bym.

Subsequently, the exposure is started (Step S109). Immediately beforethe start of exposure, the image pickup element 1 is repeatedly appliedwith a substrate application high voltage pulse VSUB of the image pickupelement 1 for forcedly releasing the charge accumulated at the photodiode 32 to a semiconductor substrate (substrate=vertical overflow drainVOFD). After this application of the high voltage pulse VSUB iscompleted and a moment when the value of VSUB is set to a value inaccordance with the number of times m for performing the time-divisionshooting is the start of exposure in Step S109.

Next, it is determined whether or not the first time-division shootingis completed (Step S110). Herein, until it is determined that the firsttime-division shooting is completed, the process is branched to J103 andstands by this time-division shooting

In this way, in Step S110, when it is determined that the firsttime-division shooting is completed, a high voltage transfer pulse isapplied to a transfer gate arranged between the photo diode 32 of theimage pickup element 1 and the vertical transfer CCD 33, therebyshifting the charge accumulated at the photo diode 32 to the verticaltransfer CCD 33 of the image pickup section 1a. This charge shifted tothe vertical transfer CCD 33 is then transferred to the verticaltransfer CCD 34 of the accumulation section 1 b.

Now, with reference to FIGS. 12 and 13, effective areas of the imagepickup element will be described. FIG. 12 shows an effective area in asuperposition relation of three time-division images and FIG. 13 showsan effective area in the image pickup area of the image pickup element.

First of all, FIG. 12 shows a case where a time-division image 51 and atime-division image 52, and a time-division image 53 are shot in atime-division manner in the stated order and a positional relation amongthe images is deviated as shown in the drawing while the images arearranged so that the same subject in each of the images positioned atthe same position. In such a case, an effective area 54 that issynthesized after the blurring compensation in the end is formed of anarea where three of the time-division images 51, 52, and 53 are alloverlapped one another. Then, the effective area 54 becomes narrower asthe amount of blurring of the images becomes larger.

In usual cases, it suffices that a predetermined area previously set onthe basis of the long side and the short side of the image (for example,an area of 98% of length and width of the image) is defined as aneffective area 56 in an image pickup area 55 (refer to FIG. 13) andother areas are not set as the effective area. However, depending on askill or the like of a photographer, expectedly large blurring mayoccur. In view of the above, after the first time-division shooting iscompleted, it is determined whether or not an absolute value Bx of theamount of blurring in the X direction from the start position ofexposure or an absolute value By of the amount of blurring in the Ydirection from the start position of exposure is equal to or larger thana previously set predetermined value α (the predetermined value a isobtained through modification into a value with which an image at leastincluding the effective area 56 as the synthesized image can be securedwhen both the absolute values Bx and By are smaller than α (Step S111).

Then, in Step S111, when it is determined that at least one of theabsolute value Bx of the amount of blurring in the X direction and theabsolute value By of the amount of blurring in the Y direction is equalto or larger than the predetermined value α, the loop of thetime-division exposure is escaped, and after a flag FLG is set as 1(Step S112), the process is shifted to a reading out process on an imagein Step S119 which will be described later (J104). This process isperformed because when the amount of blurring is equal to or larger thanthe predetermined value α, the effective area of the image in which theblurring is properly compensated becomes narrower than the previouslyset predetermined area, and to prevent this situation, thistime-division shooting is ended and the effective area with thepredetermined size is to be secured.

After that, in Step S112, when FLG is set as 1, it is meant that thenumber of times for performing the time-division shooting does not reachthe predetermined times m. The overall exposure time does not reachT_(Exp) and the level of the synthesized image is lower than theappropriate level. Therefore, at this time, as will be described above,the image is amplified by the gain controller amplifier 3. For example,if the number of times for performing the time-division shooting is setto k (k is a positive integer smaller than m), the amplification gain ofthe gain controller amplifier 3 is m/k times of the normal case.

On the other hand, in Step S111, when it is determined that both theabsolute value Bx of the amount of blurring in the X direction and theabsolute value By of the amount of blurring in the Y direction aresmaller the predetermined value α, next, the pixel value synthesisprocess is performed (Step S113). In the pixel value synthesis, as hasbeen already described above, after the first image in the verticaltransfer CCD 34 and the second image held in the horizontal transfer CCD35 obtained through the time-division shooting are shifted in therespective transfers CCD 34 and 35 to compensate the blurring on thebasis of the amounts of blurring in the X and Y directions which arecalculated on the basis of the output signals from the angular ratesensors 19 and 20, the synthesis process is conducted. The pixel valuesynthesis process is conducted at a higher speed than the exposure timeΔT_(Exp) in the time-division shooting in usual cases.

Next, n+1 is stored in the memory [n] that stores the number of times nfor performing the time-division shooting already performed (Step S114).

After that, it is determined whether or not the number of times n forperforming the time-division shooting already performed is equal to theset number of times m for performing the time-division shooting (StepS115).

Herein, when the relation of n=m is not achieved, the process isbranched to J103, and the time-division shooting process and the pixelvalue synthesis process are repeatedly performed as described above.

In Step S115, when it is determined that the relation of n=m isestablished, the process is branched to J104, and the synthesized imagestored in the horizontal transfer CCD 35 is transferred to the verticaltransfer CCD 34 and thereafter the reading out process to be describedlater in Step S119 is conducted.

On the other hand, when it is determined in Step S106 that the relationof T_(Exp)<T_(Limit) is established, no substantial blurring occurswithout performing the time-division shooting. Therefore, at this time,the exposure time T_(Exp) is stored in the memory [ΔT_(Exp)] (StepS116).

Next, the exposure is started (Step S117), and it is determined whetheror not the exposure time reaches the exposure time ΔT_(Exp) stored inthe memory [ΔT_(Exp)] Then, when it is determined the exposure timereaches the exposure time ΔT_(Exp), the charge generated by the photodiode 32 is shifted to the vertical transfer CCD 33, thereby ending theexposure (Step S118). After this exposure is ended, the charge of thevertical transfer CCD 33 in the image pickup section 1 a is transferredto the vertical transfer CCD 34 of the accumulation section 1 b. On theother hand, until the exposure time reaches the exposure time ΔT_(Exp),the above-mentioned determination is repeatedly performed.

In this manner, when the exposure time T_(Exp) is shorter than theblurring limit exposure time T_(Limit), this time-division shooting isnot performed. As the image without blurring is outputted by oneshooting similarly to the conventional shooting, the process becomesfacilitated and it is possible to prevent wasteful electric powerconsumption due to the blurring compensation.

When it is determined in Step S118 that the exposure is ended, when itis determined in Step S112 that the flag FLG is completed to be set as1, or when it is determined in Step S115 that the relation of n=m isestablished, the shot image is read out to the outside via the verticaltransfer CCD 34 and the reading horizontal transfer CCD 37 from theimage pickup element 1 (Step S119).

The image signal thus read out from the image pickup element 1 isprocessed by the CDS 2 and amplified by the gain controller amplifier(AMP) 3. After that, the image signal is converted into a digital signalby the A/D converter 4 (Step S120). It should be noted that when theflag FLG is set as 1 in Step S112, the amplification gain of the gaincontroller amplifier (AMP) 3 is m/k times of the normal amplificationgain as described above.

After that, the image data obtained through the digitalization of theimage signal is stored in the buffer memory 8 a and then subjected to apredetermined signal process by the information processing section 8(Step S121).

The signal process by the information processing section 8 includes aprocess for extracting an image with a previously set area in which itis assumed that the blurring compensation has been effectively conductedfrom the image data that is outputted from the image pickup element 1.That is, the information processing section 8 has also a function of theeffective area extraction section. As has been described above, in theimage pickup element 1, in order that an effective area in which an areawhere all the time-division images are overlapped one another ispreviously set (the effective area 56 shown in FIG. 13) is included, thesynthesis process is only performed on the time-division image in whichthe amount of blurring with the first shot time-division image as thereference is smaller than the previously set predetermined value α.Therefore, the information processing section 8 can easily perform theeffective area extraction process.

Next, the image data subjected to the signal process by the informationprocessing section 8 is temporarily stored in the DRAM 9 (Step S122).

After that, the image data stored in the DRAM 9 is compressed by thecompress/expand section 10 (Step S123) and the compressed image data isrecorded in the recording medium 11 (Step S124), thereby ending thisprocess.

Subsequently, FIG. 11 is a flowchart showing another process examplecorresponding to the image pickup element 1 of the second configurationexample shown in FIG. 7 when an image is picked up by a digital camera.It should be noted that in FIG. 11, a description of a part in which thesame process as that in FIG. 10 is performed is appropriately omittedwhile mentioning that effect.

The processes from Steps S201 to S207 are the same as those from StepsS101 to S107 in FIG. 10. It should be noted that in Step S201, when thefirst release switch 18 a is not in the closed state, the jumpingdestination of the process is J201, and in Step S205, when the secondrelease switch 18 b is not in the closed state, the jumping destinationof the process is J202.

Then, after the process in Step S207 is completed, next,(β×T_(Limit))/ΔT_(Exp) is calculated, and the thus calculated value isstored in the memory [k] that stores the variable k (Step S208). β inthe calculation formula is a previously set predetermined coefficientand β×T_(Limit) that is a numerator of the calculation formularepresents the overall amount of blurring necessary for securing theeffective area with the predetermined size or larger (hereinafterreferred to as “allowable amount of blurring”).

Herein the effective area means an area where all the time-divisionimages are overlapped one another. Therefore, if this time-divisionshooting is performed by more than {(β×T_(Limit))/ΔT_(Exp)} times forthe exposure time ΔT_(Exp), the overall amount of blurring exceeds theallowable amount of blurring.

Then, it is determined whether or not a relation of k>m is established(Step S209).

Herein, when it is determined that the relation of k>m is established,that is, k is larger than the number of times m for performing thetime-division shooting that is previously set as the initial value andwhen it is determined that the amount does not exceed the allowableamount of blurring even if this time-division shooting is performed by mtimes, m is stored in the memory [p] that stores the number of times forperforming the time-division shooting (Step S210).

On the other hand, when it is determined that the relation of k>m is notestablished and when the number of times for performing thetime-division shooting exceeds k times, this time-division shooting theoverall amount of blurring exceeds the allowable amount of blurring andk is stored in the memory [p] (Step S211).

When the process in Step S210 or S211 is ended, next, the initial value0 is stored in the memory [n] that stores the number of times forperforming the time-division shooting (Step S212).

Then, the exposure is started (Step S213), and until the exposure timebecomes ΔT_(Exp), the exposure is continued (Step S214).

Next, the pixel value synthesis is performed as in the process in StepS113 of FIG. 10 (Step S215) and n+1 is stored in the memory [n], wherebythe number of times for performing the time-division shooting isincremented (Step S216).

Subsequently, it is determined whether or not the number of times forperforming the time-division shooting [n] is p (Step S217), and when itis determined that the relation of n=p is not established, the processjumps to J203 and the time-division shooting and the pixel valuesynthesis are repeatedly performed as described above.

Then, in Step S217, when it is determined that the relation of n=p isestablished, the process jumps to J204 and the synthesized image storedin the horizontal transfer CCD 35 is transferred to the verticaltransfer CCD 34 and then read out to the outside of the image pickupelement 1 via the reading horizontal transfer CCD 37 (Step S221).

In Step S223 thereafter, the image read out from the image pickupelement 1 is amplified by m/p times the normal amplification gain by thegain controller amplifier 3. Such an amplification is conducted becausein order to obtain the image at the appropriate level, thistime-division shooting needs to be performed by m times, but the actualnumber of times for performing the time-division shooting is p, and theamplification needs to be performed in accordance with the number oftimes Also, in Step S206, the processes in Steps S218 to S226 in thecase where it is determined that the relation of T_(Exp)<T_(Limit) isestablished are the same as those in Steps S116 to S124 of FIG. 10, andtherefore the description thereof will be omitted.

As described in the above, in the process described with reference toFIG. 11, a value of β times the blurring limit exposure time T_(Limit)obtained on the basis of empirical knowledge is defined as the allowableoverall amount of blurring to secure an area equal to or larger than thepreviously set effective area, and the control is performed on thenumber of times for performing the time-division shooting so that theamount does not exceed the set amount of blurring. Therefore, there is amerit of facilitating the control as compared to the process describedin FIG. 10 for determining whether or not the overall amount of blurringexceeds the allowable value through real time monitoring on the amountof blurring. Then, as the process shown in FIG. 11 is conducted, theprovision of the angular rate sensors 19 and 20 and the A/D converters21 and 22, etc. is unnecessary, whereby the configuration is alsosimplified.

It should be noted that in the above-mentioned flowcharts in FIGS. 10and 11, the operations of the digital camera corresponding to the imagepickup element 1 according to the second configuration example shown inFIG. 7 is described. Also, the operations of the digital cameracorresponding to the image pickup element 1 according to the firstconfiguration example shown in FIG. 6 is the same except that the pixelvalue synthesis process in step S113 of FIG. 10 is only replaced by thepixel value synthesis in the image pickup element 1 corresponding toFIG. 6 which has been already described.

According to the first embodiment, the amount of blurring from thefirstly shot time-division image among the plurality of continuouslyshot time-division images is synthesized to a time-division image havingthe previously set amount of blurring within the predetermined amountrange, whereby the synthesized image with the predetermined effectivearea can be generated. In this way, even for an unspecified number ofphotographers with different photographing skills, the image effectivearea can have a size equal to or larger than the predetermined size.

Then, when the signal level of the synthesized image does not reach thepredetermined level, the synthesized image is amplified so as to reachthe predetermined level, whereby it is possible to obtain the image withthe appropriate exposure.

In addition, when the process shown in FIG. 11 is performed, the numberof continuous shooting can be controlled without relying on a particularpart such as an angular rate sensor for detecting the amount ofblurring, whereby the configuration can be made simpler and also thecontrol becomes easier.

Second Embodiment

FIGS. 14A to 14F and 15 show a second embodiment of the presentinvention. FIGS. 14A to 14F are timing charts for showing operations ofthe image pickup element 1. In the second embodiment, the same parts asthose of the first embodiment are denoted by the same reference numeralsand a description thereof will be omitted. Mainly, different points willbe only described.

According to the second embodiment, the images obtained through thetime-division shooting are read out from the image pickup element 1 anddigitalized, and stored in the buffer memory 8 a, and then the blurringcompensation and the image synthesis are performed by the informationprocessing section 8.

The electric configuration of this digital camera is basicallysubstantially the same as that shown in FIG. 1, and with reference toFIG. 1 and the like, different points from the first embodiment will beonly described.

First of all, in the second embodiment, the solid-state image pickupelement 1 is not the image pickup element that can performed theblurring compensation described in the first embodiment but is asolid-state image pickup element used in a normal digital camera, forexample, a CCD solid-state image pickup element. It should be noted thatnot only the CCD solid-state image pickup element but also a CMOSsolid-state image pickup element or the like may be of course used.

The information processing section 8 includes the buffer memory 8 a asdescribed above. The buffer memory 8 a is adapted to have a capacitywith which a plurality of images obtained through this time-divisionshooting can be stored. Then, the information processing section 8calculates to find out the amount of mutual blurring among the pluralityof images obtained through this time-division shooting on the basis ofthe control of the CPU 7, and on the basis of the found amount ofblurring the image process is performed to compensate the blurring ofthe plurality of images before synthesis. Herein, the informationprocessing section 8 calculates the amount of blurring of the image onthe basis of a few featured points in the image as will be describedabove. Therefore, the information processing section 8 doubles as theblurring detection section. In this way, according to the secondembodiment, the angular rate sensors 19 and 20 and the A/D converters 21and 22 shown in FIG. 1 may not be necessarily provided.

Next, operations of the digital camera according to this embodiment willbe described. First of all, with reference to FIGS. 14A to 14F,operations of the image pickup element 1 will be described.

As shown in FIG. 14A, when a shooting trigger signal is generated as thesecond release switch 18 b is in the closed state, a clock signal CLK issupplied from the timing generator 5 to the image pickup element 1 asshown in FIG. 14B.

The image pickup element 1 that receives the clock signal CLK isrepeatedly applied with a substrate application high voltage pulse VSUBfor forcedly releasing the charge accumulated at the photo diode thatconstitutes the pixel of the image pickup element 1 to a semiconductorsubstrate (substrate=vertical overflow drain VOFD) as shown in FIG. 14E.When the application of the high voltage pulse VSUB is ended, theexposure is started.

When the predetermined exposure time of the time-division shootingΔT_(Exp) is ended, a shift pulse TP for shifting the charge at the photodiode of the image pickup element 1 to the vertical transfer CCD isoutputted as shown in FIG. 14F. After that, in synchronism with avertical synchronism signal VD shown in FIG. 14C, the respectivetransfer electrodes are applied with voltages Vφ1 to Vφ4 as shown inFIG. 14D, whereby the image is read out from the image pickup element 1.

In addition, in synchronism with the start of reading of the image fromthe image pickup element 1, as shown in FIG. 14E, the image pickupelement 1 is applied again with VSUB for a predetermined applicationtime Tsub.

When this VSUB application is ended, the exposure is started in asimilar manner. After that, in synchronism with the next verticalsynchronism signal VD shown in FIG. 14D, the reading of an image throughthe second time-division shooting is conducted.

The operations described above are performed by a predetermined numberof times (for example, 10 times). Then, as is apparent from thisdescription, a time obtained by subtracting the VSUB application timeTsub from a reading time Tread becomes the exposure time of thetime-division shooting ΔT_(Exp).

The time-division image read out from the image pickup element 1 issubjected to reset noise removal by the CDS 2 and then analog signalamplification is performed by the gain controller amplifier (AMP) 3.Herein, the amplification gain of the gain controller amplifier (AMP) 3is set to A1×A2 when an amplification gain in accordance with an ISO(International Organization for Standardization) sensitivity Sv is setas A1 and an amplification gain for making up the shortage amount ofexposure for the image obtained through this time-division shooting isset as A2.

It should be noted that when the amount of exposure in the normalshooting is set as E1 and the amount of exposure obtained through thistime-division shooting is set as B2, a relation of A2=E1/E2 isestablished. Furthermore, to be specific, when the exposure time forobtaining the appropriate exposure is set as T_(Exp) and thetime-division shooting is performed by m times due to the exposure timeT_(Exp)/m that is obtained by equally dividing the above-mentioned valueby m times, A2=T_(Exp)/(T_(Exp)/m)=m is established in eachtime-division shooting.

The amplified analog image signal amplified by the gain controlleramplifier (AMP) 3 is converted into a digital signal by the A/Dconverter 4 in accordance with a signal supplied from the TG 5.

The image signal that is the digital signal converted by the A/Dconverter 4 is processed by the information processing section 8,thereby generating image data.

Next, FIG. 15 is a flowchart showing a compensation process for ablurred image in the information processing section 8.

When this process is started, first of all, as an initial setting, 0 isstored in the memory [i] for storing the variable i corresponding to anID for identifying a time-division image (Step S301).

Next, a blurring Δ (i, i+1) between an image I (i) and an image I (i+1)is calculated (Step S302). The blurring Δ is found out by setting a fewfeatured points in the image 1 (i) and performing a known motion vectorcalculation on featured points of the image 1 (i+1) which are identicalto the above-mentioned featured points to obtain the relative deviationin the corresponding positions. It should be noted that the blurring Δis a vector.

After that, a scalar value |ΣΔ(k, k+1)| of ΣΔ(k, k+1) subjected tointegration (subsequent vector addition) of the blurring Δ (where k=0 toi) is calculated, and this value and a previously set predeterminedvalue α are compared with each other (Step S303). Herein, when it isdetermined that the relation of |ΣΔ(k, k+1)|>α is not established, thecontent of the memory [i] is incremented (Step S304).

Subsequently, on the basis of the blurring Δ, the blurring between theimage 1 (i) and the image 1 (i+1) is compensated, and thereafter thecorresponding pixel values are added (synthesized) (Step S305).

Next, i and m−1 are compared with each other (Step S306). Herein, mdenotes the number of times for performing the time-division shooting,that is, the number of pixels obtained through this time-divisionshooting. In Step S306, when it is determined that the relation of i=m−1is not established, the process is branched to J301 where theabove-mentioned processes are repeatedly performed.

On the other hand, in Step S306, when it is determined that the relationof i=m−1 is established, next, the average value of the synthesizedimages is calculated (Step S307).

In Step S303, when it is determined that the relation of |ΣΔ(k, k+1)|>αis established, the overall amount of blurring during this time-divisionshooting is determined to be larger than the allowable value (at whichit is difficult to secure the effective area), the process is branchedto J302, and the above-mentioned process in Step S307 is conducted. Inthis way, after the process in Step S307 is completed, this process isended.

As described in the above, the information processing section 8 excludesfrom the synthesis process target a time-division image having theamount of blurring larger than a previously set predetermined value withthe first shot time-division image as the reference so that an areawhere all time-division images are overlapped one another falls within apreviously set effective area.

As regard to the time-division image having the amount of blurringwithin the predetermined value, the image in the previously set area isextracted from the first shot time-division image among the synthesizedimages. With this configuration, the extraction process for theeffective area equal to or larger than the predetermined size can beeasily conducted.

In this way, the image data of the effective area in which the blurringis compensated by the information processing section 8 is temporarilystored in the DRAM 9 and thereafter compressed into image data of apredetermined format such as JPEG by the compress/expand section 10before being recorded in the recording medium 11.

It should be noted that in the above description, the blurring detectionis performed on the basis of the featured points in the time-divisionimage but the blurring detection may be of course performed on the basisof an output from an angular rate sensor or the like.

According to the second embodiment, substantially the same effects asthose of the first embodiment are achieved and it is possible togenerate the synthesized image in which the blurring is compensated onthe basis of the time-division image within the predetermined amount ofblurring among the time-division images read out from the image pickupelement 1.

In addition, in the digital camera or the like that uses the normalimage pickup element 1 without having the blurring compensation functionas well, the blurring compensation can be conducted.

Furthermore, the blurring detection is performed on the basis of thefeatured points in the time-division image, whereby the provision of anangular rate sensor or the like is unnecessary and the configuration canbe made simpler.

Having described the preferred embodiments of the invention referring tothe accompanying drawings, it should be understood that the presentinvention is not limited to those precise embodiments and variouschanges and modifications thereof could be made by one skilled in theart without departing from the spirit or scope of the invention asdefined in the appended claims.

1. An electronic blurring compensation device, comprising: a shootingsection for continuously shooting a plural images; a detection sectionfor detecting blurring of the images; a number-of-shooting controlsection for controlling the number of continuous shootings such that atotal amount of blurring over of a predetermined number of continuouslyshot images among the plurality of the continuously shot images fallswithin a previously set predetermined value; a blurring compensationsection for compensating mutual blurring of the plurality of images shotby the predetermined times on the basis of the control ofnumber-of-shooting control section; and an image synthesize section forsynthesizing the images compensated by the blurring compensationsection.
 2. The electronic blurring compensation device according toclaim 1, wherein: the shooting section is structured by including animage pickup element having a plurality of pixels arranged in matrix anda register for holding a first image generated by the pixels; theblurring compensation section shifts relative positions of the firstimage held in the register and a second image generated by the pixelssuch that relative blurring between the second image and the first imageis cancelled; and the image synthesize section synthesizes the firstimage and the second image which are compensated by the blurringcompensation section.
 3. The electronic blurring compensation deviceaccording to claim 1, wherein the number-of-shooting control sectioncontrols the number of shootings such that a total amount of blurring ofcontinuously shot images falls within a previously set predeterminedvalue.
 4. The electronic blurring compensation device according to claim1, wherein the number-of-shooting control section controls the number ofshootings such that a total exposure time of the plurality of thecontinuously shot images is equal to or lower than an exposure timeobtained by multiplying an inverse number of a focal length of ashooting lens by a previously set predetermined coefficient.
 5. Theelectronic blurring compensation device according to claim 2, furthercomprising an amplification section for amplifying the image synthesizedby the image synthesize section when the number of the continuously shotimages does not reach the previously set predetermined number.
 6. Theelectronic blurring compensation device according to claim 1, wherein:the shooting section is structured by including an image pickup elementhaving a plurality of pixels arranged in matrix, a signal processingsection for generating a plurality of images digitalized by receiving anoutput signal from the image pickup element, and a memory section forstoring the image processed by the signal processing section; and theblurring compensation section compensates blurring of the image storedin the memory section.